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(54) Title: THE PRODUCTION OF AUGMENTED-REALITY AUDIO 




(57) Abstract: A method for the delivery of augmented-reality audio in- 
cludes creating an occlusion (12) in an auditory canal (14) of a listener so as 
to limit the transmission of external sounds in the range of human hearing 
from the environment to at least one eardrum of the listener. An acoustic 
sensor (20) is arranged within the auditory canal distally of the occlusion 
(12) and the sensor (20) is used to sense the external sounds as they arrive 
at the sensor (20). An acoustic actuator (22) is arranged within the auditory 
canal proximally of the occlusion (12). Auxiliary input audio signals are re- 
ceived from an external source to be rendered as virtual audio to the listener. 
The actuator (22) is used to deliver an output audio signal to the eardrum 
(18), the output audio signal constituting at least one of input audio signals 
corresponding to the external sounds, the auxiliary input audio signals and 
combinations of the foregoing. 
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"The production of augmented-reality audio" 

Field of the Invention lAKOItfiTOFFf 3 1 JAN 

The present invention relates to the production of augmented-reality audio* 
More particularly, the invention relates to a method , of, and equipment for, the 
5 production of augmented-reality audio and , more specifically, for r^dering virtual 
spatial audio concurrently with the real sound environment in such a manner that a 
listener clearly perceives both the real and virtual sound sources as localized within the 
listener's phonal auditory space. 

10 Background to the Invention 

The Applicant is aware of various techniques for producing virtual spatial audio 
as well as for delivering electronically-produced audio to the ears such that the real 
sound environment remains audible and accessible, but not necessarily properly 
spatialised, when the electronically-produced audio is played back. Still further, the 

15 Applicant is aware of the use of earphones, commonly referred to as "open 
tubephones", that sit near the opening of the ear canal, but do not block the canal's 
entrance, and allow the presentation of both virtual and real sound sources such that 
both are properly spatialised within the listener's personal auditory space. 

Some of the techniques for producing virtual spatial audio apply electronic 

20 filtering to the sound signals for the left and right ear using filters that are designed to 
simulate the acoustic filtering of the listeners' outer ears. These techniques include 
accurately estimating the filter fiinctions of the listener's outer ears for any location in 
space around the listener in a manner which generates for the listener a virtual soxmd 
field that is nearly indistinguishable firom the real sound field. 

25 Some of the techniques for producing virtual spatial audio use "closed" 

earphones to acoustically deUver the filtered signals for the left and right ear to the ears 
of the listener. Closed earphones, such as circumaural headphones that fit around the 
outer ear, or ear-bud headphones that fit in the ear, are referred to as closed earphones 
because they significantly occlude the sound path to the ear, thereby substantially 

30 distorting the external sound input to the ear. These types of earphones significantiy 
reduce the quality of the external sound field that is heard by the listener. Thus, a 
disadvantage of the techniques that use a closed earphone is that the closed earphones 
that deliver the virtual spatial audio also attenuate and/or distort sounds in the real, 
external environment 

35 Yet other techniques for producing virtual spatial audio use "open" earphones. 

Some open earphones provide a low acoustic impedance seal (a partial seal) around the 
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outer ear and thereby produce less occlusion for external sounds tiian closed earphones. 
A disadvantage of these techniques that use open earphones with a low acoustic 
impedance seal is that the low acoustic impedance seal can still distort tiie acoustic 
filtering of the outer ear. This distortion disrupts the normal acoustic spatialisation cues 
S associated with outer ear filtering and results in poor acoustic spatialisation of external 
sound sources. 

Still other techniques for producing virtual spatial audio use "open" tubephones 
that sit near, but do not block, the ear canal. Hiese earphones allow the presentation of 
both vktual and real sound sources such that both are properly spatialised within the 

10 listener's personal auditory space. A disadvantage of these techniques is that one does 
not have fiiU control of the sound signal presented to the listener. More specifically, 
one cannot modify the intensity level or characteristics of the real sound sources. 
Another disadvantage is the relatively high power required to drive the open 
tubephones because of the distance tiiey sit away from the eardrum. A fiirfher 

1 5 disadvantage is that the relatively cumbersome and delicate nature of the placement of 
the tubephones detracts from the general utility of such devices. 

Summary of the Inveiition 

According to a first aspect of the invention, there is provided a method for the 
20 delivery of augmented-reality audio, the method including the steps of; 

creating an occlusion in an auditory canal of a listener so as to limit the 
transmission of external sounds in the range of human hearing from the environment to 
at least one eardrum of the listener; 

arranging an acoustic sensor within the auditory canal distally of the occlusion 
25 and using the sensor to sense the external sounds as they arrive at the sensor; 

arranging an acoustic actuator within the auditory canal proximally of the 
occlusion; 

receiving auxiliary input audio signals to be rendered , as virtual audio to the 
listener; and 

30 using the actuator to deliver an output audio signal to flie eardrum, the output 

audio signal constituting at least one of input audio signals corresponding to the 
external sounds, tiie auxiliary input audio signals and combinations of the foregoing. 

In this specification the term "distally of the occlusion" means on the side of the 
occlusion facing outwardly from the ear and, conversely, the term "proximally of the 

35 occlusion" means on the side of the occlusion facing towards the eardrum. 



wo 2005/004534 



PCT/AU2004/000902 



3 

The method may include selecting the acoustic sensor so as to minimise 
disraption to acoustic filtering of an outer ear of the listener. Further, the method may 
include implementing the sensor as a microphone with an input signal frequency range 
comparable to that of nomial human hearing, i.e. with a frequency range of about 50 
S Hz to 16 kHz. In this way, flie acoustic sensor may be operable to capture the real 
sound field. 

The method may include implementing the acoustic actuator as a miniature 
loudspeaker, of miniature loudspeaker array, with a frequency range comparable to that 
of normal human hearing. 

10 The method may include processing the auxiliary audio signals using methods 

of virtual auditory space to create the illusion for liie listener that sounds representative 
of the auxiliary audio signals originate at specific locations in the listener's personal 
auditory space around the listener's head. 

Further, the meliiod may include using signal processing to modify and mix any 

15 combination of input audio signals to produce the output audio signal fed to the 
acoustic actuator. Thus, the method may include processing the input audio signals 
corresponding to the extemal sounds to one of amplifying and attenuating the input 
audio signals corresponding to the extemal sounds. In addition, the method may 
include processing at least one of the input audio signals corresponding to the extemal 

20 sounds and the auxiliary audio signals to perform enhancemmt of at least one class of 
sounds of interest within the extemal sounds. For example, the method may include 
processing the input audio signals to perform speech enhancement. 

Still further, the method may include processing the input audio signals 
corresponding to the extemal sounds to effect active noise cancellation. Also, the 

25 method may include processing the input audio signals corresponding to the extemal 
sounds to assist in protecting the listener's hearing by rapidly attenuating loud extraial 
sounds. 

In addition, the method may include playing back the output audio signal 
through the acoustic actuator with a time delay after sensing of the input audio signals 
30 corresponding to the extemal sounds by the sensor. 

The method may include transmitting the input audio signals corresponding to 
the extemal soimds to an extemal devices before or after processing for further 
processing, delayed playback to the listener, or playback to other listeners. 

Further, the method may include using a device controller to control operation 
35 of at least certain components, the components being the sensor, the acoustic actuator 
and the signal processor. 
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Additionally, the method may include transmitting audio signals to other devices 
and/or receiving signals from other devices. 

The method may also include receiving information and/or instructions from the 
other devices, such as a wearable computer or other devices, in regard to how the 
S auxiliary audio signals are to be rendered so as to generate acoustic characteristics such 
as the spatial location or the motion trajectory of the object in extra-personal space, the 
loudness of the auditory object, the timbre of the auditory object, the ambient acoustic 
characteristics associated with a virtual environment of an auditory object. Thus, the 
method may include manipulating audio filters used to generate the auxiliary input 
1 0 audio signals to modify the auditory object in virtual auditory space. 

Preferably, the method is implemented at both ears of the listener, but may only 
be performed at one of the listener's ears. 

According to a second aspect of the invention, there is provided equipment for 
the deliv^ of augmented-reality audio, the equipment including: 
15 an occluder for occluding an auditory canal of a listener so as to limit the 

transmission of external sounds in the range of human hearing from the environment to 
at least one eardrum of the listener; 

a sensor for sensing the external sounds, the sensor being arranged on one side 
of the occlusion to lie within the auditory canal distally of the occlusion; 
20 an acoustic actuator arranged on an opposed side of the occlusion so as to lie in 

the auditory canal proximally of the occlusion; and 

a receiver for receiving auxiliary input audio signals from a source of the 
auxiliary input audio signals to be rradered as virtual audio to the listoier, the receiver 
and the sensor being in communication with the actuator so that the actuator is operable 
25 to deliver an output audio signal to the eardrum, the output audio signal constituting at 
least one of input audio signals corresponding to the external sounds, the auxiliary 
input audio signals and combinations of the foregoing. 

The sensor may be configured to minimise dismption to acoustic filtering of an 
outer ear of the listraer. Thus, the sensor may be in the form of a microphone having a 
30 frequency range comparable to that of normal human hearing. 

The actuator may be in the form of a loudspeaker with a frequency range 
comparable to that of normal human hearing. 

The equipment may include a signal processor for processing the input audio 
signals corresponding to the external sounds and the auxiliary input audio signals prior 
35 to the signals being received by the actuator. 



wo 2005/004534 



PCT/AU2004/000902 



5 

The signal processor may modify and mix any combination of input audio 
signals to produce the output audio signal fed to the acoustic actuator. Further, the 
signal processor may effect one of amplification and attenuation of the input audio 
signals corresponding to the external sounds. Also, the signal processor processes at 

S least one of the input audio signals corresponding to the external sounds and the 
auxiliary audio signals to perform enhancement of at least one class of sounds of 
interest within the external sounds. In addition, the signal processor may process the 
input audio signals corresponding to the external sounds to effect active noise 
cancellation. The signal processor may further process the input audio signals 

10 corresponding to the external sounds to assist in protecting the listmer's hearing. 

The signal processor may include a time delay for playing back the output audio 
signal through the acoustic actuator with a time delay after sensing of the input audio 
signals corresponding to the external sounds by the sensor. 

The equipment may include an amplifier for amplifying signals including the 

15 signals fi-om tiie sensor, a signal output from the signal processor and the auxiliary 
input audio signals. 

The equipment may include an external transmitter device for transmission of 
the input audio signals corresponding to the external sounds sensed by the sensor. 

The equipment may include a device controller for enabling the listener to 
20 control operation of at least certain components, the components being the sensor, the 
acoustic actuator and the signal processor. 

The receiver may be configured to be received in the auditory canal. More 
particularly, the receiver may be contained in the occluder. The receiver may be in 
wireless communications with the source by an appropriate wireless communications 
25 protocol such as radio fi-equency, infra red, ultrasonic, or the like. It will also be 
appreciated that, instead of a wireless link, the receiver could be wired to the source. 

The receiver may form part of a transceiver to enable signals to be transmitted to 
other devices. 

The equipment may include an adjustable audio filter arrangement used to 
30 enable the auxiliary input audio signals to be manipulated to modify an auditory object 
in virtual auditory space. 

Further, the equipment may include an energy storage unit to supply power. 
Preferably, the equipment is configured for each ear of the listener. 
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Brief Description of the Drawings 

The invention is now described by way of example with reference to the 
accompanying drawings in which:- 

Figure 1 shows a schematic representation of equipment, in accordance with an 
5 embodiment of the invention, for the delivery of augmented-reality audio in use; 
Figure 2 shows a schematic, block diagram of the equipment; and 
Figure 3 shows a front view of another version of the equipment 

Detailed Description of Exemplary Embodiment 

10 In the drawings, reference numeral 10 generally designated equipment, in 

accordance with an embodiment of the invention, for the delivery of augmented-reality 
audio. 

The equipment 10 includes an occluder 12 receivable in an auditory canal 14 of 
an ear 16 of a listener. The occluder 12 is shaped and configured effectively to block 

15 the auditory canal 14 and to inhibit sounds reaching an eardrum 18 of the listener. 
More particularly, the occluder 12 reduces, by about 10 dB, the transmission of sounds 
in the range of human hearing from the enviroranent to the eardrum 18. In the 
embodiment shown in Figure 1 of the drawings, the occluder 12 is individually 
moulded to fit in a bony part of the auditory canal 14 of the listener. The occluder 12 

20 can be made of any suitable hard, malleable compound or soft compound or a mixture 
of the two. 

A sensor 20 is mounted on a distal side of the occluder 12 to face an entrance to 
the auditory canal 14. The sensor 20 senses sounds from the extemal environment 
within the range of human hearing. Thus, for example, the sensor 20 is implemented as 
25 a microphone having an input signal frequency range of about 50 Hz to 16 kHz. The 
sensor 20 senses the sounds from the extemal environment and generates input audio 
signals correspondiog to the extemal sounds. For ease of explanation, these input audio 
signals will be referred to below as the extemal audio input signals. 

An acoustic actuator 22 is arranged on a distal side of the occluder 12. It is 
30 therefore positioned in the auditory canal 14 and delivers sounds to the eardrum 18 of 
the listoier, as will be described in greater detail below. The acoustic actuator 22 is 
implemented in the form of a miniature loud speaker or miniature loud speaker array 
which has an audible frequency range comparable to that of normal human hearing. 

A signal processor 24 is contained within a body of the occluder 12, as shown in 
35 greater detail in Figure 2 of the drawings. The signal processor 24 processes incoming 
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signals prior to transmission of the signals to the actuator 22, An amplifier 26 is 
interposed between the signal processor 24 and the actuator 22. 

The equipment 10 further includes a receiver constituting part of a transceiver 
28 contained within the body of the occluder 12, The transceiver 28 conmiunicates 
5 wirelessly with a source 30 of auxiliary, input audio signals. For example, the soiirce 
30 is a cellular telephone or some other device. The transceiver 28 communicates with 
the source 30 via any suitable wireless communications protocol, more particularly, a 
suitable low power wireless connection. The wireless connection can take the form of 
a radio-frequency signal conforming to an accepted standard such as 801.1 lb. Instead, 
10 the wireless connection could be an infra red connection using infrared as a carrier 
signal or the wireless connection could be an ultrasonic connection using an \iltrasonic 
carrier signal. Instead of a wireless cormection, the transceiver 28 could be wired to the 
source 30 with the source 30, for example, being wom behind the ear 16 of the 
listener.. 

15 The equipment 10 also includes an auxiliary conmnmications device 32 carried 

or wom by the listener. The device 32 allows devices with multiple transmission 
protocols to communicate with the transceiver 28 either wirelessly or in a wired 
manner. 

Still further, the equipment includes an external device 34. The external device 
20 34 receives signals sent from the parts of the equipment 10 contained in the auditory 
canal 14 of the listener for onward transmission to other devices (not shown). The 
device 34 communicates with the sensor 20 via flie transceiver 28 or auxiliary 
communications devices 32. 

The equipment 10 also includes a controller 36 for controlling operation of the 
25 equipment. The controller 36 communicates with the source 30 as well as with a data 
reading unit 38 for input of information to the controller 36. The controller 36 is 
accessible by the listener for controlling operation of the eqmpment 10. 

For ease of explanation, the parts of flie equipment 10 contained within the 
auditory canal 14 shall be referred to below as an earphone 40. It will be appreciated, 
30 that, in use, the equipment 10 comprises two such earphones 40, one for each auditory 
canal 14 of the listener for raabling sounds to be delivered to both eardrums 18 of tiie 
listens 10. 

In the embodiment shown in Figure 1 of the drawings, the occluder 12 of the 
earphone 40 is a mouldable, individualized element moulded specifically for the 
35 auditory canal 14 of a particular listener. The occluder 12 fits snugly within the bony 
portion of the auditory canal 14 and it is possible to wear it semi-permanentiy. 
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In another, more generalised version of the equipment 10, the earphones 40 are 
carried on stalks 42 projecting radially inwardly from a holder 44 wom on a user's 
head. Other components of the equipment 10 are carried in a holder 46 on the holder 
44. In this embodiment, flie equipment 10 approximates a conventional set of 
5 headphones but with the modification that the earphones 40 are still received 
completely in the auditory canal 14 effectively to block the auditory canal 14. 

Referring again to Figure 2 of the drawings it is to be noted that the equipment 
10 includes an energy storage unit or battery 48 contained in the body of the occluder 
12. The battery 48 could, for example, be a re-chargeable battery. 

10 The equipment 10 is intended for delivery of augmented-reality audio to a 

listener. Accordingly, in use, an earphone 40 is placed completely in each auditory 
canal 14 of the listener. As indicated above, the earphones 40 effectively occlude the 
auditory canal 14 and substantially inhibit the passage of external sounds from the 
environment to the eardrums 1 8 of the listener. 

15 It is to be noted that, by having the earphones 40 completely within the auditory 

canals 14 of the listener, the acoustic filtering of the outer ear 16 of the listener is 
preserved to enable external sounds to be detected by each sensor 20. In this way, 
monaural spectral cues that are derived from the filtering by the outer ear 16 are 
available for sound localisation and generation to enable the listener to have a precise 

20 perception of the external auditory environment. 

By having the earphones 40 wittiin the auditory canals 14 and the proximity of 
the acoustic actuator 22 to its associated eardrum 18, the power required to generate 
sounds at an appropriate sound level is reduced, reducing the power consumption of the 
equipment 10 and minimising drain on the battery 48. In addition, acoustic leakage 

25 around the earphones 40 is iiiininused. 

Using the embodiment shown in Figure 3 of the drawings, a less deep placement 
of the earphone 40 within each auditory canal 14 may result and less individualised 
methods of fitting the earphones 40 may need to be employed. Although these factors 
may reduce the utility of the equipment 10 slightly it is compmsated for by the ease 

30 and convenience of the processes required for fitting, inserting and removing the 
earphones 40. 

The earphones 40 are designed to preserve the frill range of hearing frequencies 
for human listeners in the frequency range from about 50 Hz to 16 kHz. The earphones 
40 are also able to transmit external sounds collected by the sensors 20 to the actuators 
35 22 as the external audio input signals and then to the eardrum without substantial 
distortion of modification. 
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It will, however, be appreciated that, by appropriate manipulation of the signal 
processor 24 sounds emanating from the environment can be transformed in a variety 
of ways. These transformations include, but are not limited to: compensating, using a 
predicted or measured transfer function for the auditory canal 14, for Ihe loss of 

5 auditory canal resonance resulting from the occlusion of the auditory canal 14 by the 
earphone 40; filtering sounds from the extemal environment so as to compensate for 
the occlusion effect produced by the occluda: 12; producing rapid changes in the gain 
of the extemal sound fliat can be triggered by the incoming sound level or by a signal 
from some extemal device commxmicated to the eaiphone 40 via the transceiver 28 or 

10 the communications device 32; producing signals that result in active noise cancellation 
; at the eardrum 18; or producing changes in the gain of specific classes of sounds based 
on their spectral-temporal characteristics (such as speech) while varying the gain 
independentiy for other sounds discriminated in terms of their different spectral- 
temporal characteristics. 

1 5 However, the main purpose of the signal processor 24 is to combine the extemal 

audio input signals from the sensor 20 and the auxiliary, input audio signals from the 
transceiver 28 to feed these combined signals to the acoustic actuator 22. The acoustic 
actuator 22 emits a composite signal constituting the extemal audio input signals from 
the sensor 20 and the auxiliary, input audio signals from the transceivor 28 to the 

20 eardrum 18. 

The controller 36 adjusts and/or changes audio filters used to create auditory 
objects in virtual auditory space. The unit 38 comprises a memory card reader, 
smartcard reader or other, data reading device for input to the controller 36 of 
information such as the audio filters required to render auditory objects in virtual 
25 auditory space. 

The source 30 communicates with the earphones 40 either via the transceiver 28 
or the auxiliaiy communications device 32. As indicated above, this conununications 
between the source 30 and the transceiver 28 or the communications device 32 is, 
preferably, effected wirelessly. 
30 The signal processor 24 receives the audio input from the transceiver 28, 

possibly in addition with information from the controller 36. The signal processor 24 
mixes the auxiliary, input audio signals from the transceiver 28 with the extemal audio 
input signals from the sensor 20 to effect augmented-reality audio made available to the 
listener. 

35 Using the Applicant's methods for virtual auditory space as disclosed in the 

Applicant's International Patent Application, Publication No. WO 01/54453 entitied 
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"The generation of customised three-dimensional sound effects for individuals", the 
contents of which are incorporated herein by reference, the additional input is 
processed by the signal processor 24 so that, when delivered to the listener via the 
actuator 22, an auditory object or objects is/are rendered for the perception of the 
5 listener at some phantom or virtual location or locations in space around the listener. 

The source 30 could be a wearable computer or some other intelligent device 
tiiat takes input from the listener or from o&er sources in regard to how the signal 
might be rendered so as to generate various characteristics of the auditory object 
These characteristics include, but are not limited to, the spatial location or the motion 

10 trajectory of the object in extra-personal space, the loudness of the auditory objection, 
its timbre or the ambient acoustic characteristics associated with the virtual 
environment of the object such as reverberance. The source 30 can perform the 
filtering required to render sounds for the right and left earphones 40. Still further, the 
source 30, optionally, sends only the signal to be rendered or the signal with tokens, 

15 coefficients and other information for the signal processor 24 to enable the signal 
processor 24 to perform the necessary filtering. Still further, the listener could select an 
information channel from the source 30 and, using the controller 36, select the 
characteristics to be rendered for the information channel received by the earphones 40. 
A common example of this is the audio channel from a mobile conmmnications device 

20 such as a cellular telephone which is then rendered by the signal processor 24 in the 
earphones 40 to appear at a particular location in auditory space as selected by the 
listener. In this case, the coeflBdents necessary to carry out such filtering are loaded 
into the signal processor 24 via the controller 36 and the unit 38. 

The sensors 20 of the earphones 40 could also be used as input devices for 

25 . secondary communications and other intelligent devices. Hence, the utterances of the 
listener are encoded by the sensors 20 and then transmitted to the device 34 using either 
the transceiver 28 or the auxiliary communications device 32, as tiie case may be. In 
tiie latter case, only low powered transmission is required betwerai the earphones 40 
and file auxiliary conamunications device 32. This, in turn, reduces current drain on flie 

30 battery 48 of the earphones 40. 

Thus, by means of the equipment 10, both the real external sound field and the 
virtual auditory field can be appreciated simultaneously and with high fidelity. On the 
one hand, sounds from the real external sound field can be passed without distortion 
from the input side of the earphone 40 to the output side and thence to the eardrum 18. 

35 In this way, the earphone 40 generates the same pattern of sound waves at the eardrum 
18 that would have occurred in the absence of the earphone 40. On the other hand. 
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using the methods of virtual auditory space, the rendered input can be combined with 
the external audio input signals by the signal processor 24 in the earphone 40. When 
delivered to the eardrums 18 of the right and left ears this results in a high fidelity 
rendering of virtual auditory space that can be appreciated in conjunction and 

5 simultaneously with the appreciation of the real extemal sound field. 

Another advantage of the invention is that the equipment 10 provides for the 
introduction of rendered signals firom a variety of sources. This can include, but is not 
limited to, a cellular telephone or other commxmications device, voice or other sound 
interfaces firom intelligent devices and music or other sounds from entertainment 

10 devices. The link to the signal processor 24 in the earphones 40 also allows listener 
control over the characteristics of these inputs such as, but not limited to, their virtual 
location, loudness and tonal quality. As the output of each earphone 40 is ahnost 
entirely sealed firom the real extemal sound field, these sounds can also be appreciated 
by the listener and not heard by other listeners near the listener using the equipment 1 0. 

15 A fijrther advantage of the invention arises firom the ability of human listeners to 

parse or separate multiple concurrent sources of information based on differences iii 
their locations in space around the listener. This is referred to as the "cocktail party 
effect" and a common example is the ability of listeners to attend to one conversation in 
a situation where there are multiple talkers and to switch attention firom one 

20 conversation to another at will. For example, if speech is rendered into a specific 
virtual location near the listener then the listener is able to selectively attend to the 
rendered talker or to ignore fiie rendered talker as desired The equipment 10 enhances 
this ability of the listener using the equipment 10. 

Yet another advantage of the invmtion is tiiat the virtual auditory object can be 

25 used to render information where the location of a particular sound may convey 
information at the same time as the listener is engaged in other activities. An example 
of such an application is in providing verbal or other spatial directions using a 
navigational guidance system for buildings or other terrain. 

Still a fiarther advantage of the invention is that hearing protection fimctions can 

30 also be managed using the equipment 10. In environments with high background noise 
that is either relatively continuous or impulsive, the gain of the earphones 40 can be 
dynamically and rapidly adjusted to significantiy alter the overall gain of the sound 
reaching the eardrums 18 of the listener. This derives in part from the fact that the 
earphone can be tightiy sealed in its associated auditory canal 14 at a location close to 

35 the eardrum 18 and, when the sound transmitting fimction is attenuated or turned off, 
will provide a very effective occlusive barrier to sound firom the real extemal sound 
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field to the eardrum. In addition, the signal processor 24 in each earphone 40 can also 
modify the sound delivered to the eardrum 18 so that, for instance, by inverting its 
phase and adding an appropriate short delay, the sound will actively cancel sound that 
may leak from the real extOTial sound field to the eardrum from aroxmd or through the 
5 earphone. 

An additional advantage of the invention is that hearing enhancement fimctions 
for the hearing-impaired listener can also be implemented in the earphones 40 using the 
signal processor 24 of each earphone 40. This can include variable gain fimction, 
background noise cancellation, amplitude compression or amplification and speech 
10 enhancement. 

The equipment 10 exploits the completely-in-the-canal wearing/fitting 
configuration and produces no perceptually relevant acoustical distortion of the normal 
filtering properties of the outer ear. Of note, these properties are directionally sensitive 
and distortions cannot be compensated for without having prior knowledge as to the 

15 direction of the incoming sound. The distortion of the transmission properties of the 
auditory canal 14 produced by occluding the canal can be compensated for using a 
single gain frinction in the signal processor 24 that approximates the resonance gain of 
the canal. Changes to tiie overall input impedance of the outer ear resulting from the 
occlusion of the auditory canal 14 by the earphone 40 may result in changes in the 

20 filtering characteristics of the outer ear (for instance ino-eased gain of the resonant 
modes of the concha - the cavity at the entrance of the ear canal). However, those 
skilled in the art will recognize that virtual auditory objects rendwed using filter 
functions recorded by microphones occluding the canal are of high fidelity and do not 
degrade the appreciation of, or the performance of tasks reliant on, the spatial 

25 information so rendered. 

It will be appreciated by persons skilled in the art that numerous variations 
and/or modifications may be made to the invention as shown in the spedfic 
embodiments without departing from the spirit or scope of the invention as broadly 
described. The present embodiments are, therefore, to be considered in all respects as 

30 illustrative and not restrictive. 
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CLAIMS: 

1 . A method for the delivery of augmented-reality audio, the metiiod including the 
stq)S of: 

creating an occlusion in an auditory canal of a list^er so as to limit the 
5 transmission of extemal sounds in the range of human hearing from the environment to 
at least one eardrum of the listener; 

arranging an acoustic sensor within the auditory canal distally of the occlusion 
and using the sensor to sense the extemal soimds as they arrive at the sensor; 

arranging an acoustic actuator within tiie auditory canal proximally of the 
10 occlusion; 

receiving auxiliary input audio signals to be rendered as virtual audio to the 
listener; and 

\ising the actuator to deliver an output audio signal to the eardrum, the output 
audio signal constituting at least one of input audio signals corresponding to the 
1 5 extemal sounds, the auxiliary input audio signals and combinations of the foregoing. 

2. The method of claim 1 which includes selecting the acoustic sensor so as to 
minimise dismption to acoustic filtering of an outer ear of the listener. 

* 

20 3. The method of claim 1 or claim 2 which includes implementing the sensor as a 
microphone with an input signal frequency range comparable to tihat of normal human 
hearing. 

4. The method of any one of the preceding claims which includes implementing 
25 the acoustic actuator as a miniature loudspeaker with a frequency range comparable to 

that of normal human hearing. 

5. The method of any one of the preceding claims which includes processing the 
auxiliary audio signals using methods of virtual auditory space to create the illusion for 

30 the listener that sounds representative of the auxiliary audio signals originate at specific 
locations in the listener's personal auditory space around the listener's head. 

6. The method of any one of the preceding claims which includes using signal 
processing to modify and mix any combination of input audio signals to produce the 

35 output audio signal fed to the acoustic actuator. 
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7. The method of claim 6 which includes processing the input audio signals 
corresponding to the external sounds to one of amplifying and attenuating the input 
audio signals corresponding to the extemal sounds. 

5 8. The method of claim 6 or claim 7 which includes processing at least one of tiie 
input audio signals corresponding to the extemal sounds and the auxiliary audio signals 
to perform enhancement of at least one class of sounds of interest within the extemal 
sounds. 

10 9. The method of any one of claims 6 to 8 which includes processing the input 
audio signals corresponding to the extemal sounds to effect active noise cancellation. 

10. The method of any one of claims 6 to 9 which includes processing the input 
audio signals corresponding to the extemal sounds to assist in protecting the listener's 

15 hearing. 

1 1 . The method of any one of the preceding claims which includes playing back the 
output audio signal through the acoustic actuator with a time delay after sensing of the 
input audio signals corresponding to the extemal sounds by the sensor. 

20 

12. The method of any one of the preceding claims which includes transmitting the 
input audio signals corresponding to the extemal sounds to an extemal device. 

13. The method of any one of flie preceding claims which includes using a device 
25 controller to control operation of at least pertain components. 

14. The method of any one of the preceding claims which includes transmittmg 
audio signals to other devices. 

30 1 5. The method of any one of the preceding claims which includes receiving signals 
from other devices. 

16. The method of claim 15 which includes receivmg information and/or 
instractions from the other devices in regard to how the auxiliary audio signals are to be 
35 rendered so as to generate acoustic characteristics associated with a virtual environment 
of an auditory object. 
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17. The method of claim 16 which includes manipulating audio filters \ised to 
generate the auxiliaiy input audio signals to modify the auditory object in virtual 
auditory space. 

5 

18. The method of any one of the preceding claims which is implemented at both 
ears of the listener. 

19. Equipment for the delivery of augmrated-reality audio, the equipm^t 
10 including: 

an occluder for occluding an auditory canal of a listmer so as to limit the 
transmission of external sounds in the range of human hearing firom the environment to 
at least one eardrum of the listener; 

a sensor for sensing the external sounds, the sensor being arranged on one side 
15 of the occlusion to lie within the auditory canal distally of the occlusion; 

an acoustic actuator arranged on an opposed side of the occlusion so as to lie in 
the auditory canal proximally of the occlusion; and 

a receiver for receiving auxiliary input audio signals firom a source of the 
auxiliary input audio signals to be rendered as virtual audio to the listener, the receiver 
20 and the sensor being in communication with flie actuator so that the actuator is operable 
to deliver an output audio signal to the eardrum, the output audio signal constituting at 
least one of input audio signals corresponding to the external sounds, the auxiliary 
input audio signals and combinations of the foregoing. 

25 20. The equipment of claim 19 in which the sensor is configured to minimise 
disruption to acoustic filtering of an outer ear of the listener. 

21. The equipment of claim 19 or claim 20 in which the sensor is in the form of a 
microphone having a firequency range comparable to that of normal human hearing. 

30 

22. The equipment of any one of claims 1 9 to 2 1 in which the actuator is in the form 
of a loudspeaker with a frequency range comparable to that of normal human hearing. 

23. The equipment of any one of claims 19 to 22 which includes a signal processor 
35 for processing the input audio signals corresponding to the external sounds and the 

auxiliary input audio signals prior to the signals being received by the actuator. 
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24. The equipment of claim 23 in which the signal processor modifies and mixes 
any combination of input audio signals to produce tiie output audio signal fed to the 
acoustic actuator. 

5 

25, The equipment of claim 23 or claim 24 in which tiie signal processor effects one 
of amplification and attenuation of the input audio signals corresponding to the external 
sounds. 

10 26. The equipment of any one of claims 23 to 25 in which the signal processor 
processes at least one of the input audio signals corresponding to the external sounds 
and the auxiliary audio signals to perform enhancement of at least one class of sounds 
of interest within the external sounds. 

15 27. The equipment of any one of claims 23 to 26 in which the signal processor 
processes the input audio signals corresponding to the external sounds to effect active 
noise cancellation. 

28. The equipment of any one of claims 23 to 27 in which the signal processor 
20 processes the input audio signals corresponding to the external sounds to assist in 

protecting the listener's hearing. 

29. The equipment of any one of claims 23 to 28 in which the signal processor 
includes a time delay for playing back the output audio signal through the acoustic 

25 actuator with a time delay after sensing of the input audio signals corresponding to the 
external sounds by the sensor, 

30. The equipment of any one of claims 23 to 29 which includes an amplifier for 
amplifying signals including the signals firom the sensor, a signal output from tiie signal 

30 processor and the auxiliary input audio signals. 

31. The equipment of any one of claims 19 to 30 which includes an external 
transmitter device for transmission of the input audio signals corresponding to the 
external soxmds sensed by the sensor. 
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32. The equipment of any one of claims 19 to 3 1 which includes a device controller 
for enabling the listener to control operation of at least certain components. 

33. The equipment of any one of claims 19 to 32 in which the receiver is configured 
5 to be received in the auditory canal. 

34. The equipment of claim 33 in which the receiver is contained in the occluder. 

35. The equipment of any one of claims 19 to 34 in which the receiver is in wireless 
1 0 communications with flie source, 

36. The equipment of any one of claims 1 9 to 35 in which the receiver forms part of 
a transceiver to enable signals to be transmitted to other devices. 

1 5 37. The equipment of any one of claims 19 to 36 which includes an adjustable audio 
filter arrangement used to enable the auxiliary input audio signals to be manipulated to 
modify an auditory object in virtual auditory space. 

38. The equipment of any one of claim 19 to 37 which includes an energy storage 
20 unit to supply power. 

39. The equipment of any one of claims 19 to 38 which is configured for each ear of 
the listener. 
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